The present invention is in the field of image and video processing, and may be usable for implementing desktop and mobile virtual studio applications (e.g., video conferencing and video calling).
Virtual studios are widely used in the high end professional TV broadcast industry. Such real time applications often use professional TV cameras, illumination setups and chroma keying (e.g., green screen processing techniques) for differentiating between foreground (FG) objects (e.g., humans, furniture) and scene background (BG), for illumination nonuniformity correction, and suchlike. Generally, foreground figures and objects are rendered as photographed in real time, while backgrounds are overlaid by graphic information, video streams and advertisements in real time or near real time. Recorded footage is often also post processed offline by post production processes in preparing said material for broadcast.
Live processing of video shots in real time is a difficult and resource consuming task, particularly due to the vast amount of data that needs to be processed within very short time periods. There been several attempts to provide techniques for processing video streams in real time. The following patent publications suggest possible solutions for real time processing of video images.
U.S. Pat. No. 7,660,463 discloses a technique for extraction of a portion of a scene without requiring excessive user interaction. In this publication the extraction is achieved by using iterated graph cuts. The method used includes segmenting an image into a foreground portion and a background portion (e.g., where an object or desired portion to be extracted is present in the foreground portion) and determining the properties corresponding to the foreground and background portions of the image. Distributions may be utilized to model the foreground and background properties. The properties may be color in one implementation and the distributions may be a Gaussian Mixture Model in another implementation. The foreground and background properties are updated based on the portions, and the foreground and background portions are updated based on the updated foreground and background properties.
U.S. Pat. No. 6,288,703 suggests a technique wherein each observed background color in an image being displayed on a monitor is identified using a cursor to select the dominant colors. Each foreground color is also identified by using the cursor as a selector. For each pixel, several candidate mattes are computed one for each background color. Each candidate matte is computed from a single background color and the set of selected foreground colors using any known method. Of the several candidate mattes, the greater of these candidates becomes the matte signal for a given pixel. If this matte is 1.0, then the pixel is part of the background. If the matte is zero, then the pixel is part of the subject. If the matte is between 1.0 and 0.0, then the pixel is in a transition area and the contribution of the background to the pixel is that pixel's matte level. A “processed foreground” is generated by removing the contribution of the background from each pixel in the image, resulting in the subject appearing against a black field. The generated candidate mattes permit the generation of processed foreground candidates, of which the smallest is the final processed foreground for a given pixel.
U.S. Pat. No. 6,271,890 suggests using a chromokeying panel, including a plurality of first and second locations, in conjunction with a chromakeying system. Each of the first and the second locations has a color within the color of a color keying spectrum of the chromakeying system. In addition, each of the first and the second locations are arranged to define a coded pattern for a computation of a perspective of the chromakeying panel with respect to a camera independently in each video field produced by the camera. The coded pattern may include a plurality of straight lines that intersect to define vertices. The plurality of straight lines may be spaced apart from each other so that for a given vertex, a ratio of the distance of one of the lines extending from the given vertex to a first adjacent vertex, to the distance of another of the lines from the given vertex to a second adjacent vertex is unique.
In US Patent publication No. 2003/152285 suggests a techniques in which chrominance, color saturation and color intensity of an actual pixel (P1) are approximated directly from associated YUV values (alpha, a, b) which leads to the avoidance of time-consuming calculations. The recognition of rapid deviations in illumination carried out in the YUV color space is based upon the approximation of a chrominance difference by an angle difference (alpha1-alpha2) of the pixels (P1, P2) to be compared, preferably in a plane (U, V) of the YUV color space. This proceeds on the assumption that the chrominance of a pixel at the occurrence of shadows and brightenings remains constant in spite of varying color saturation and color intensity. The suggested method may be supplemented by a rapid decision program including additional decision parameters which excludes complex calculations of angle operations and separation error, even at significant deviations in illumination.
U.S. Pat. No. 5,764,306 suggests a method allowing real-time replacement of the designated background portion of an incoming video signal with an alternate background. The method utilizes the actual background image for reference as the basis for determining the background and foreground elements within the image with the end result being comparable to traditional blue-screen processes, such as chroma-key and ultimate technology, but requiring only a personal computer, video camera and the software. In this case, however, the reference background image can be any reasonably static scene with a sufficient and stable light source captured by the camera. The video data stream is modified in real-time by comparisons against the reference background image and is then passed on to its original destination. Multiple signal-noise processing algorithms are applied in real-time against the signal to achieve a visually acceptable matte.